Rotating LIDAR with co-aligned imager

ABSTRACT

Example implementations are provided for an arrangement of co-aligned rotating sensors. One example device includes a light detection and ranging (LIDAR) transmitter that emits light pulses toward a scene according to a pointing direction of the device. The device also includes a LIDAR receiver that detects reflections of the emitted light pulses reflecting from the scene. The device also includes an image sensor that captures an image of the scene based on at least external light originating from one or more external light sources. The device also includes a platform that supports the LIDAR transmitter, the LIDAR receiver, and the image sensor in a particular relative arrangement. The device also includes an actuator that rotates the platform about an axis to adjust the pointing direction of the device.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/592,541, filed Oct. 3, 2019, which is a continuation of U.S. patentapplication Ser. No. 15/671,845, filed Aug. 8, 2017. The foregoingapplications are incorporated herein by reference.

BACKGROUND

Active sensors, such as light detection and ranging (LIDAR) sensors,radio detection and ranging (RADAR) sensors, and sound navigation andranging (SONAR) sensors, among others, can scan an environment byemitting signals toward the environment and detecting reflections of theemitted signals. Passive sensors, such as image sensors and microphonesamong others, can detect signals originating from sources in theenvironment.

An example LIDAR sensor can determine distances to environmentalfeatures while scanning through a scene to assemble a “point cloud”indicative of reflective surfaces. Individual points in the point cloudcan be determined, for example, by transmitting a laser pulse anddetecting a returning pulse, if any, reflected from an object in theenvironment, and then determining a distance to the object according toa time delay between the transmission of the pulse and the reception ofits reflection. Thus, a three-dimensional map of points indicative oflocations of reflective features in the environment can be generated.

An example image sensor can capture an image of a scene viewable to theimage sensor. For instance, the image sensor may include an array ofcharge-coupled devices (CCDs) or other types of light sensors. Each CCDmay receive a portion of light from the scene incident on the array.Each CCD may then output a measure of the amount of light incident onthe CCD during an exposure time when the CCD is exposed to the lightfrom the scene. With this arrangement, an image of the scene can begenerated, where each pixel in the image indicates one or more values(e.g., colors, etc.) based on outputs from the array of CCDs.

SUMMARY

In one example, a device includes a light detection and ranging (LIDAR)transmitter that emits light pulses toward a scene according to apointing direction of the device. The device also includes a LIDARreceiver that detects reflections of the emitted light pulses reflectingfrom the scene. The device also includes an image sensor that capturesan image of the scene according to the pointing direction of the deviceand based on external light originating from one or more external lightsources. The device also includes a platform that supports the LIDARtransmitter, the LIDAR receiver, and the image sensor in a particularrelative arrangement. The device also includes an actuator that rotatesthe platform about an axis to adjust the pointing direction of thedevice.

In another example, a vehicle comprises a light detection and ranging(LIDAR) sensor that includes a transmitter and a receiver. Thetransmitter emits light pulses toward a scene according to a pointingdirection of the LIDAR sensor. The receiver detects reflections of theemitted light pulses propagating from the scene. The vehicle alsocomprises a camera that captures an image of the scene according to apointing direction of the camera and based on external light originatingfrom one or more external light sources. The vehicle also comprises aplatform that supports the LIDAR sensor and the camera in a particularrelative arrangement. The vehicle also comprises an actuator thatrotates the platform about an axis to simultaneously change the pointingdirection of the LIDAR sensor and the camera according to the particularrelative arrangement.

In yet another example, a method involves scanning a scene using a lightdetection and ranging (LIDAR) sensor. The LIDAR sensor emits lightpulses toward the scene and detects reflections of the emitted lightpulses from the scene. The method also involves generating an image ofthe scene using an image sensor that detects external light originatingfrom one or more external light sources. The method also involvesrotating a platform that supports the LIDAR sensor and the image sensorin a particular relative arrangement. Rotating the platform comprisessimultaneously rotating the LIDAR sensor and the image sensor about anaxis.

In still another example, a system comprises means for scanning a sceneusing a light detection and ranging (LIDAR) sensor. The LIDAR sensor mayemit light pulses toward the scene and detect reflections of the emittedlight pulses from the scene. The system also comprises means forgenerating an image of the scene using an image sensor that detectsexternal light originating from one or more external light sources. Thesystem also comprises means for rotating a platform that supports theLIDAR sensor and the image sensor in a particular relative arrangement.Rotating the platform may comprise simultaneously rotating the LIDARsensor and the image sensor about an axis.

These as well as other aspects, advantages, and alternatives will becomeapparent to those of ordinary skill in the art by reading the followingdetailed description with reference where appropriate to theaccompanying drawings. Further, it should be understood that thedescription provided in this summary section and elsewhere in thisdocument is intended to illustrate the claimed subject matter by way ofexample and not by way of limitation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified block diagram of a system that includesco-aligned rotating sensors, according to example embodiments.

FIG. 2 illustrates a device that includes co-aligned rotating sensors,according to example embodiments.

FIG. 3A illustrates another device that includes co-aligned rotatingsensors, according to example embodiments.

FIG. 3B is a cross-section view of the device of FIG. 3A.

FIG. 4 illustrates a partial view of yet another device that includesco-aligned rotating sensors, according to example embodiments.

FIG. 5 is a first conceptual illustration of images based on data fromone or more rotating sensors, according to example embodiments.

FIG. 6 is a second conceptual illustration of images based on data fromone or more rotating sensors, according to example embodiments.

FIG. 7 is a third conceptual illustration of images based on data fromone or more rotating sensors, according to example embodiments.

FIG. 8 is a flowchart of a method, according to example embodiments.

FIG. 9 is a simplified block diagram of a vehicle, according to anexample embodiment.

FIG. 10A illustrates several views of a vehicle equipped with a sensorsystem, according to an example embodiment.

FIG. 10B illustrates an example operation of the sensor system.

DETAILED DESCRIPTION

Exemplary implementations are described herein. It should be understoodthat the word “exemplary” is used herein to mean “serving as an example,instance, or illustration.” Any implementation or feature describedherein as “exemplary” or “illustrative” is not necessarily to beconstrued as preferred or advantageous over other implementations orfeatures. In the figures, similar symbols typically identify similarcomponents, unless context dictates otherwise. The exampleimplementations described herein are not meant to be limiting. It willbe readily understood that the aspects of the present disclosure, asgenerally described herein, and illustrated in the figures, can bearranged, substituted, combined, separated, and designed in a widevariety of different configurations.

I. OVERVIEW

Example devices, systems, and methods herein relate to synchronizedoperation of two or more sensors, such as a LIDAR sensor and an imagesensor for instance, that sense a surrounding environment. By way ofexample, sensor fusion algorithms can be employed to merge data from animage sensor and a LIDAR sensor to generate a 3D representation of ascanned environment. For instance, an example system can be configuredto map pixels in one or more images captured by an image sensor topoints in a point cloud generated using a LIDAR sensor. With thisarrangement, for instance, the 3D representation of the scannedenvironment may indicate color information determined using the imagesensor combined with distance information determined using the LIDARsensor.

In some scenarios, combining LIDAR data from a spinning LIDAR with imagedata from an image sensor may involve a variety of technical challenges.Example challenges include synchronizing the timing of photon collectionby the two sensors, synchronizing the fields-of-view of the two sensors,and managing different exposure time requirements of the two sensors,among other examples. By way of example, a LIDAR receiver may beconfigured to detect reflections of an emitted light pulse during anexposure time period of 1 nanosecond. However, in this example, exposuretimes suitable for an image sensor may range from 1 microsecond to 1millisecond depending on lighting conditions and scene content in theenvironment (e.g., daytime conditions, nighttime conditions, etc.).Additionally, if the image sensor is rotated synchronously with theLIDAR sensor (e.g., at 10 Hz), the longer exposure time period of theimage sensor may result in image artifacts such as image smearing, etc.

Accordingly, one example device of the present disclosure may include aLIDAR sensor and an image sensor mounted to a rotating platform in aparticular relative arrangement. In one implementation, the image sensormay image a scene in an environment of the device through a same lensthat focuses light toward a receiver of the LIDAR sensor. In anotherimplementation, the image sensor can be configured as part of a camerathat has a camera lens separate from a LIDAR lens that focuses lightonto the receiver of the LIDAR sensor.

Regardless of the implementation, in some examples, the image sensor canthus be rotated synchronously with the LIDAR sensor such that respectivefields-of-view (FOVs) of the two sensors remain at least partiallyoverlapping in response to the rotation. Alternatively or additionally,the image sensor can be configured to capture image pixels (e.g., byexposing charge-coupled device (CCD) elements, CMOS image sensingelements, active pixel sensor elements, or other sensing elements in theimage sensor to external light from the imaged scene) according toemission times and/or detection times of one or more light pulsesemitted by the LIDAR sensor.

Through this process, the present method may improve accuracy and/orefficiency of computer operations related to combining sensor data fromthe two sensors by synchronizing the FOVs of the two sensors and/or theimage pixel capture times with the LIDAR light pulse emission/detectiontimes, even while the two sensors are rotating to scan the surroundingenvironment. By way of example, the device may include a controller thatmaps the captured image pixels to corresponding points in the pointcloud generated using the LIDAR sensor, based on at least the co-alignedFOVs and/or matching times at which the respective data was generated bythe two sensors. Alternatively or additionally, an external or remotecomputing system can receive the sensor data from the device and performthe mapping process remotely.

In some examples, the image sensor can be implemented according to atime-delay-and-integration (TDI) configuration. For instance, the imagesensor may include an array of sensing elements (e.g., CCDs, etc.) thatprovide data for generating image pixels of a captured image.

In one implementation, the image sensor (or the controller) may shift adata frame (e.g., column of pixels) captured by a first column ofsensing elements to an adjacent data frame associated with an adjacentsecond column of sensing elements synchronously with the rotation of theplatform that supports the LIDAR sensor and the image sensor. Datacaptured by the second column of sensing elements (e.g., after arotation of the platform that causes the position of the second column(relative to the scene) to correspond or be similar to the position ofthe first column when the shifted data was captured) can then be addedto the shifted data frame. To that end, one or more columns of the arrayof sensing elements may be aligned (e.g., parallel, etc.) with the axisof rotation of the rotating platform. Further, for instance, the imagesensor (or controller) may determine a time delay between shifting thecolumns based on the rate of rotation of the platform.

In another implementation, whether or not the array of sensing elementsis aligned with the axis of rotation, the image sensor (or thecontroller) may combine (e.g., sum) data (e.g., light intensityinformation, etc.) indicated by a first image pixel captured using afirst sensing element to data indicated by a second image pixel capturedusing a second sensing element (after a time delay from capture of thefirst image pixel). For instance, the controller may determine that therotating motion of the platform causes an imaged object in a first imageto become distorted in a second image (captured after a time delay) dueto the associated change in the pointing direction (e.g., viewpoint,etc.) of the image sensor. Further, such distortion may depend onvarious factors such as lens characteristics, the position of the imagesensor relative to a camera lens, and/or a distance between the imagedobject and the device, among other factors. To that end, for instance,the controller may select the second sensing element based on the rateof rotation of the platform, distance information (e.g., from the LIDAR)of the object in the first image pixel relative to the device (e.g.,expected pixel location of the imaged object in the second image),and/or other optical characteristics of the device (e.g., focal lengthof a lens that focuses light onto the image sensor, position of imagesensor relative to lens, etc.).

Regardless of the implementation, the image sensor can be configured tocapture multiple image pixels representing a particular (same) region ofthe scene at different times as the device rotates. Further, the imagesensor (or the controller) can combine detections of the particularregion of the scene (i.e., combine image pixels captured at differenttimes during the rotation) according to the apparent motion of the scenerelative to the device as the device rotates.

Through this process, improved sensor data quality (e.g., reduced imagesmearing and/or other image artifacts associated with the apparentmotion of the scene relative to the image sensor during a long exposuretime) can be achieved by reducing the exposure times of the sensingelements during individual image (or pixel) captures while the platformis rotating, and combining the individual images (or pixels) toeffectively achieve a desired (longer) exposure time.

Additionally, with this arrangement for instance, improved mapping(e.g., sensor fusion, etc.) of sensor data from the LIDAR device and theimage sensor can be achieved by synchronizing collection, in the timedomain and/or the space domain, of the data from the image sensor andthe data from the LIDAR sensor. For instance, the controller of thedevice may synchronize image pixel capture times with LIDAR light pulseemission and/or detection times, even if the time period between LIDARlight pulse emissions is less than a suitable exposure time for theimage sensor.

II. EXAMPLE SENSORS

Although example sensors described herein include LIDAR sensors andcameras (or image sensors), other types of sensors are possible as well.A non-exhaustive list of example sensors that can be alternativelyemployed herein without departing from the scope of the presentdisclosure includes RADAR sensors, SONAR sensors, sound sensors (e.g.,microphones, etc.), motion sensors, temperature sensors, pressuresensors, etc.

To that end, example sensors herein may include an active range sensorthat emits a signal (e.g., a sequence of pulses or any other modulatedsignal) based on modulated power provided to the sensor, and thendetects reflections of the emitted signal from objects in thesurrounding environment. Alternatively or additionally, example sensorsherein may include passive sensors (e.g., cameras, microphones,antennas, pressure sensors, etc.) that detect external signals (e.g.,background signals, etc.) originating from external source(s) in theenvironment.

Referring now to the figures, FIG. 1 is a simplified block diagram of asystem 100 that includes co-aligned rotating sensors, according to anexample embodiment. As shown, system 100 includes a power supplyarrangement 102, a controller 104, a LIDAR 106, a camera 108, a rotatingplatform 110, one or more actuators 112, a stationary platform 114, arotary link 116, a housing 118, and a display 140.

In other embodiments, system 100 may include more, fewer, or differentcomponents. For example, system 100 can optionally include one or moresensors (e.g., gyroscopes, accelerometers, encoders, microphones,RADARs, SONARs, thermometers, etc.) in addition to or instead of LIDAR106 and camera 108. Additionally, the components shown may be combinedor divided in any number of ways. For example, LIDAR 106 and camera 108can alternatively be implemented as a single physical component thatincludes one or more of the components shown in LIDAR 106 and camera108. Thus, the functional blocks of FIG. 1 are illustrated as shown onlyfor convenience in description. Other example components, arrangements,and/or configurations are possible as well without departing from thescope of the present disclosure.

Power supply arrangement 102 may be configured to supply, receive,and/or distribute power to various components of system 100. To thatend, power supply arrangement 102 may include or otherwise take the formof a power source (e.g., battery cells, etc.) disposed within system 100and connected to various components of system 100 in any feasiblemanner, so as to supply power to those components. Additionally oralternatively, power supply arrangement 102 may include or otherwisetake the form of a power adapter configured to receive power from one ormore external power sources (e.g., from a power source arranged in avehicle to which system 100 is mounted, etc.) and to transmit thereceived power to various components of system 100.

Controller 104 may include one or more electronic components and/orsystems arranged to facilitate certain operations of system 100.Controller 104 may be disposed within system 100 in any feasible manner.In one embodiment, controller 104 may be disposed, at least partially,within a central cavity region of rotary link 116. In anotherembodiment, one or more functions of controller 104 can be alternativelyperformed by one or more physically separate controllers that are eachdisposed within a respective component (e.g., LIDAR 106, camera 108,etc.) of system 100.

In some examples, controller 104 may include or otherwise be coupled towiring used for transfer of control signals to various components ofsystem 100 and/or for transfer of data from various components of system100 to controller 104. Generally, the data that controller 104 receivesmay include sensor data based on detections of light by LIDAR 106 and/orcamera 108, among other possibilities. Moreover, the control signalssent by controller 104 may operate various components of system 100,such as by controlling emission and/or detection of light by LIDAR 106,controlling image capture by camera 108, and/or controlling actuator(s)112 to rotate rotating platform 110, among other possibilities.

To that end, in some examples, controller 104 may include one or moreprocessors, data storage, and program instructions (stored in the datastorage) executable by the one or more processors to cause system 100 toperform the various operations described herein. In some instances,controller 104 may communicate with an external controller or the like(e.g., a computing system arranged in a vehicle, robot, or othermechanical device to which system 100 is mounted) so as to helpfacilitate transfer of control signals and/or data between the externalcontroller and the various components of system 100.

Additionally or alternatively, in some examples, controller 104 mayinclude circuitry wired to perform the various functions describedherein. Additionally or alternatively, in some examples, controller 104may include one or more special purpose processors, servos, or othertypes of controllers. For example, controller 104 may include aproportional-integral-derivative (PID) controller or other control loopfeedback apparatus that operates actuator(s) 112 to modulate rotation ofrotating platform 116 according to a particular frequency or phase.Other examples are possible as well.

LIDAR sensor 106 may include any device configured to scan a surroundingenvironment by emitting light and detecting reflections of the emittedlight. To that end, as shown, LIDAR 106 includes a LIDAR transmitter120, a LIDAR receiver 122, and one or more optical elements 124.

Transmitter 120 may be configured to transmit light (or other signal)toward an environment of system 100. In one example, transmitter 120 mayinclude one or more light sources that emit one or more light beamsand/or pulses having wavelengths within a wavelength range. Thewavelength range could, for example, be in the ultraviolet, visible,and/or infrared portions of the electromagnetic spectrum depending onthe configuration of the light sources. In some examples, the wavelengthrange can be a narrow wavelength range, such as provided by lasersand/or some light emitting diodes.

In some examples, the light source(s) in transmitter 120 may includelaser diodes, diode bars, light emitting diodes (LEDs), vertical cavitysurface emitting lasers (VCSELs), organic light emitting diodes (OLEDs),polymer light emitting diodes (PLEDs), light emitting polymers (LEPs),liquid crystal displays (LCDs), microelectromechanical systems (MEMS),fiber lasers, and/or any other device configured to selectivelytransmit, reflect, and/or emit light to provide a plurality of emittedlight beams and/or pulses.

Receiver 122 may include one or more light detectors (e.g., photodiodes,avalanche photodiodes, etc.) that are arranged to intercept and detectreflections of the light pulses emitted by transmitter 120 and reflectedfrom one or more objects in a surrounding environment of system 100. Tothat end, receiver 122 may be configured to detect light havingwavelengths in the same wavelength range as the light emitted bytransmitter 120. In this way, for instance, LIDAR 106 may distinguishreflected light pulses originated by LIDAR 106 from other light in theenvironment.

In some examples, receiver 122 may include a photodetector array, whichmay include one or more detectors each configured to convert detectedlight (e.g., in the wavelength range of light emitted by transmitter120) into an electrical signal indicative of the detected light. Inpractice, such a photodetector array could be arranged in one of variousways. For example, the detectors can be disposed on one or moresubstrates (e.g., printed circuit boards (PCBs), flexible PCBs, etc.)and arranged to detect incoming light that is traveling along theoptical path from the optical lens. Also, such a photodetector arraycould include any feasible number of detectors aligned in any feasiblemanner.

Additionally, the detectors in the array may take various forms. Forexample, the detectors may take the form of photodiodes, avalanchephotodiodes (e.g., Geiger mode and/or linear mode avalanchephotodiodes), silicon photomultipliers (SiPMs), phototransistors,cameras, active pixel sensors (APS), charge coupled devices (CCD),cryogenic detectors, and/or any other sensor of light configured toreceive focused light having wavelengths in the wavelength range of thelight emitted by transmitter 120.

In some examples, LIDAR 106 can select or adjust a horizontal scanningresolution by changing a rate of rotation of LIDAR 106 and/or adjustinga pulse rate of light pulses emitted by transmitter 120. As a specificexample, transmitter 120 can be configured to emit light pulses at apulse rate of 15,650 light pulses per second. In this example, LIDAR 106may be configured to rotate at 10 Hz (i.e., ten complete 360° rotationsper second). As such, receiver 122 can detect light with a 0.23°horizontal angular resolution. Further, the horizontal angularresolution of 0.23° can be adjusted by changing the rate of rotation ofLIDAR 106 or by adjusting the pulse rate. For instance, if LIDAR sensor106 is instead rotated at 20 Hz, the horizontal angular resolution maybecome 0.46°. Alternatively, if transmitter 120 emits the light pulsesat a rate of 31,300 light pulses per second while maintaining the rateof rotation of 10 Hz, then the horizontal angular resolution may become0.115°. Other examples are possible as well. Further, in some examples,LIDAR 106 can be alternatively configured to scan a particular range ofviews within less than a complete 360° rotation of LIDAR 106.

Optical element(s) 124 can be included in or otherwise coupled totransmitter 120 and/or receiver 122. In one example, optical element(s)124 can be arranged to direct light from a light source in transmitter120 toward the environment. In another example, optical element(s) 124can be arranged to focus and/or guide light from the environment towardreceiver 122. In yet another example, optical element(s) 124 can bearranged to filter background light incident from the surroundingenvironment from the focused light directed toward receiver 122. Assuch, optical element(s) 124 may include any feasible combination ofmirrors, waveguides, light filters, lenses, and/or any other opticalcomponents arranged to guide propagation of light through physical spaceand/or adjust certain light characteristics.

In some implementations, optical elements 124 may include at least onemirror arranged to fold an optical path between an optical lens and aphotodetector (or light detector) in receiver 122. Each such mirror maybe fixed within LIDAR 106 in any feasible manner. Also, any feasiblenumber of mirrors may be arranged for purposes of folding the opticalpath. For instance, optical elements 124 may include two or more mirrorsarranged to fold the optical path two or more times between an opticallens of LIDAR 106 and a light detector array of receiver 122.

In some implementations, optical elements 124 may include a light filterarranged to reduce or prevent light having wavelengths outside thewavelength range of the light emitted by transmitter 120 frompropagating toward receiver 122. With such arrangement for instance, thelight filter can reduce noise due to background light propagating fromthe scanned environment and originating from an external light sourcedifferent than light sources of transmitter 120.

Camera 108 may be any camera (e.g., a still camera, a video camera,etc.) configured to capture images of the environment in which system100 is located. As shown, camera 108 includes an image sensor 126 andone or more optical elements 130.

Image sensor 126 may include any imaging device that detects andprovides data indicative of an image. As shown, image sensor 126 mayinclude an arrangement of light sensing elements 128 that each provide ameasure of light waves incident thereon. To that end, sensing elements128 may include charge-coupled devices (CCDs, active pixel sensors,complementary metal-oxide-semiconductor (CMOS) photodetectors, N-typemetal-oxide-semiconductor (NMOS) photodetectors, among otherpossibilities.

Further, in some examples, data from sensing elements 128 can becombined according to the arrangement of the sensing elements 128 togenerate an image. In one example, data from a two-dimensional (2D)array of sensing elements may correspond to a 2D array of image pixelsin the image. Other examples are possible as well (e.g.,three-dimensional (3D) arrangement of sensing elements, etc.).

In some examples, a sensing element can optionally include multipleadjacent light detectors, where each detector is configured to detectlight having a particular wavelength or wavelength range. For instance,an image pixel may indicate color information (e.g., red-green-blue orRGB) based on a combination of data from a first detector that detectsan intensity of red light, a second detector that detects an intensityof green light, and a third detector that detects an intensity of bluelight. Other examples are possible as well.

In one embodiment, image sensor 126 may be configured to detect visiblelight propagating from the scene. Further, in this embodiment, receiver122 of LIDAR 106 may be configured to detect invisible light (e.g.,infrared, etc.) within a wavelength range of light emitted bytransmitter 120 of LIDAR 106. In this embodiment, system 100 (orcontroller 104) can then combine data from LIDAR 106 with data fromcamera 108 to generate a colored three-dimensional (3D) representation(e.g., point cloud) of the scanned environment.

Optical element(s) 130 may include any combination of optical componentssuch as lenses, mirrors, waveguides, light filters or any other type ofoptical component similarly to optical element(s) 124. Further, opticalelements 130 can be arranged to focus, direct, and/or adjust lightcharacteristics of incident light for propagation toward sensingelements 128. As noted above, for instance, optical elements 130 mayinclude light filters that allow wavelengths of light associated with aparticular color (e.g., red, green, blue, etc.) to propagate toward aparticular sensing element.

Although not shown, in some implementations, system 100 may include ashared optical element that is employed for both LIDAR 106 and camera108. For example, a shared lens can be arranged to focus light incidenton the shared lens toward receiver 122 of LIDAR 106 and toward imagesensor 126. For instance, optical elements 124 and/or 130 may include aselectively or partially reflective surface (e.g., dichroic mirror, halfmirror, etc.) that receives focused light from the shared lens, directsa first portion of the focused light toward receiver 122, and directs asecond portion of the focused light toward image sensor 126. Forinstance, a dichroic mirror can be positioned along a path of thefocused light, and may have dielectric properties that cause the firstportion of the focused light (having wavelengths associated with theemitted light pulses from transmitter 120) toward receiver 122, whiletransmitting the second portion of the focused light (having otherwavelengths such as visible light from the scene, etc.) toward imagesensor 126.

Thus, in some examples, fields-of-view (FOVs) of LIDAR 106 and camera108 may at least partially overlap due to the shared lens. Further,optical element(s) 124 and/or 130 may include one or more opticalelement(s) (e.g., dichroic mirrors, half mirrors, etc.) that direct afirst portion of the focused light (e.g., including reflections of thelight emitted by transmitter 120) toward receiver 122, and direct asecond portion of the focused light originating from external lightsources (e.g., including light having different wavelengths thanwavelengths of the emitted light of transmitter 120) toward camera 108.Other implementations are possible as well for simultaneously detectingexternal light (using camera 108) and reflections of LIDAR-emitted light(using receiver 122).

Further, in some implementations, system 100 may include a LIDAR lensfor focusing light onto receiver 122 and a separate camera lens forfocusing light onto image sensor 126. Additionally, in some instances,the FOVs of LIDAR 106 and camera 108 can be configured to at leastpartially overlap even if LIDAR 106 and camera 108 employ separateoptical lenses. For example, LIDAR 106 and camera 108 can be configuredin a particular relative arrangement (e.g., to have similar or samepointing directions).

Rotating platform 110 may be configured to rotate about an axis. To thatend, rotating platform 110 can be formed from any solid materialsuitable for supporting one or more components mounted thereon. Forexample, LIDAR 106 (and/or transmitter 120 and receiver 122 thereof) andcamera 108 (and/or image sensor 126 thereof) may be supported (directlyor indirectly) by rotating platform 110 such that each of thesecomponents moves relative to the environment while remaining in aparticular relative arrangement in response to rotation of rotatingplatform 110. In particular, each of these components could be rotated(simultaneously) relative to an axis so that system 100 may obtaininformation from various directions. In this manner, a pointingdirection of system 100 can be adjusted horizontally by actuatingrotating platform 110 to different directions.

In order to rotate platform 110 in this manner, one or more actuators112 may actuate rotating platform 110. To that end, actuators 112 mayinclude motors, pneumatic actuators, hydraulic pistons, and/orpiezoelectric actuators, among other possibilities.

With this arrangement, controller 104 could operate actuator 112 torotate rotating platform 110 in various ways so as to obtain informationabout the environment. In one example, rotating platform 110 could berotated in either direction. In another example, rotating platform 110may carry out complete revolutions such that LIDAR 106 (and camera 108)provides a 360° horizontal FOV of the environment. Moreover, rotatingplatform 110 may rotate at various frequencies so as to cause system 100to scan the environment at various refresh rates. In one embodiment,system 100 may be configured to have a refresh rate of 15 Hz (e.g.,fifteen complete rotations of the system 100 per second).

Stationary platform 114 may take on any shape or form and may beconfigured for coupling to various structures, such as to a top of avehicle for example. Also, the coupling of stationary platform 114 maybe carried out via any feasible connector arrangement (e.g., boltsand/or screws). In this way, system 100 could be coupled to a structureso as to be used for various purposes, such as those described herein.

Rotary link 116 directly or indirectly couples stationary platform 114to rotating platform 110. To that end, rotary link 116 may take on anyshape, form and material that provides for rotation of rotating platform110 about an axis relative to the stationary platform 114. For instance,rotary link 116 may take the form of a shaft or the like that rotatesbased on actuation from actuator 112, thereby transferring mechanicalforces from actuator 112 to rotating platform 110. In oneimplementation, rotary link 116 may have a central cavity in which oneor more components of system 100 may be disposed. In some examples,rotary link 116 may also provide a communication link for transferringdata and/or instructions between stationary platform 114 and rotatingplatform 110 (and/or components thereon such as LIDAR 106, camera 108,etc.).

Housing 118 may take on any shape, form, and material and may beconfigured to house one or more components of system 100. In oneexample, housing 118 can be a dome-shaped housing. Further, for example,housing 118 may be composed of a material that is at least partiallynon-transparent, which may allow for blocking of at least some lightfrom entering the interior space of housing 118 and thus help mitigatethermal and noise effects of ambient light on one or more components ofsystem 100. Other configurations of housing 118 are possible as well.

In some implementations, housing 118 may be coupled to rotating platform110 such that housing 118 is configured to rotate about theabove-mentioned axis based on rotation of rotating platform 110. In oneimplementation, LIDAR 106, camera 108, and possibly other components ofsystem 100 may each be disposed within housing 118. In this manner,LIDAR 106 and camera 108 may rotate together with housing 118.

In some implementations, although not shown, system 100 can optionallyinclude multiple housings similar to housing 118 for housing certainsub-systems or combinations of components of system 100. For example,system 100 may include a first housing for LIDAR 106 and a separatehousing for camera 108. In this example, LIDAR 106 and camera 108 (andtheir respective housings) may still be mounted on or otherwisesupported by rotating platform 110. Thus, rotating platform 110 canstill simultaneously rotate both sensors 106 and 108 in a particularrelative arrangement, even if sensors 106 and 108 are physicallyimplemented in separate housings. Other examples are possible as well.

Display 140 can optionally be included in system 100 to displayinformation about one or more components of system 100. For example,controller 104 may operate display 140 to display images captured usingcamera 108, a representation (e.g., 3D point cloud, etc.) of anenvironment of system 100 indicated by LIDAR data from LIDAR 106, and/ora representation of the environment based on a combination of the datafrom LIDAR 106 and camera 108 (e.g., colored point cloud, etc.). To thatend, display 140 may include any type of display (e.g., liquid crystaldisplay, LED display, cathode ray tube display, projector, etc.).Further, in some examples, display 140 may have a graphical userinterface (GUI) for displaying and/or interacting with images capturedby camera 108, LIDAR data captured using LIDAR 106, and/or any otherinformation about the various components of system 100 (e.g., powerremaining via power supply arrangement 102). For example, a user canmanipulate the GUI to adjust a scanning configuration of LIDAR 106and/or camera 108 (e.g., scanning refresh rate, scanning resolution,etc.).

As noted above, system 100 may alternatively include additional, fewer,or different components than those shown. For example, although system100 is shown to include LIDAR 106 and camera 108, system 100 canalternatively include additional co-aligned rotating sensors, and/ordifferent (e.g., RADARs, microphones, etc.) types of sensors. Further,it is noted that the various components of system 100 can be combined orseparated into a wide variety of different arrangements. For example,although LIDAR 106 and camera 108 are illustrated as separatecomponents, one or more components of LIDAR 106 and camera 108 canalternatively be physically implemented within a single device (e.g.,one device that includes transmitter 120, receiver 122, sensing elements128, etc.). Thus, this arrangement of system 100 is described forexemplary purposes only and is not meant to be limiting.

FIG. 2 illustrates a device 200 that includes co-aligned rotatingsensors, according to example embodiments. As shown, device 200 includesa LIDAR 206, a camera 208, a rotating platform 210, and a stationaryplatform 214, a LIDAR lens 224, and a camera lens 230 which may besimilar, respectively, to LIDAR 106, camera 108, rotating platform 110,and stationary platform 114, optical element(s) 124, and opticalelement(s) 130. To that end, device 200 illustrates an exampleimplementation of system 100 where LIDAR 206 and camera 208 each haveseparate respective optical lenses 224 and 230.

As shown, light beams 280 emitted by LIDAR 206 propagate from lens 224along a pointing direction of LIDAR 206 toward an environment of LIDAR206, and reflect off one or more objects in the environment as reflectedlight 290. As such, LIDAR 206 can then receive reflected light 290(e.g., through lens 224) and provide data indicating the distancebetween the one or more objects and the LIDAR 206.

Further, as shown, camera 208 receives and detects external light 292.External light 292 may include light originating from one or moreexternal light sources, background light sources (e.g., the sun, etc.),among other possibilities. To that end, external light 292 may includelight propagating directly from an external light source toward lens 230and/or light originating from an external light source and reflectingoff one or more objects in the environment of device 200 beforepropagating toward lens 230. As such, for example, camera 208 maygenerate an image of the environment based on external light 292. Theimage may include various types of information such as light intensitiesfor different wavelengths (e.g., colors, etc.) in the external light290, among other examples.

Further, as shown, camera 208 is coupled to LIDAR 206 (e.g., mounted ontop of LIDAR 206) in a particular relative arrangement (e.g., similarpointing directions toward a left side of the page). As such, in someexamples, the fields-of-view (FOVs) of LIDAR 206 and camera 208 may atleast partially overlap due to their similar respective pointingdirections, even if there is an offset (e.g., vertical offset, etc.)between the exact respective positions of sensors 206 and 208. It isnoted that other arrangements are also possible (e.g., camera 208 can bealternatively mounted to a bottom side or different side of LIDAR 206,LIDAR 206 and camera 208 may have different pointing directions, etc.).

Further, as shown, rotating platform 210 mounts LIDAR 206, and thussupports LIDAR 206 and camera 208 in the particular relative arrangementshown. By way of example, if rotating platform 210 rotates, the pointingdirections of LIDAR 206 and camera 208 may simultaneously changeaccording to the particular relative arrangement shown. In turn, anextent of overlap between the respective FOVs of LIDAR 106 and camera208 may remain unchanged in response to such rotation by platform 210.

In some examples, LIDAR 206 (and/or camera 208) can be configured tohave a substantially cylindrical shape and to rotate about an axis ofdevice 200. Further, in some examples, the axis of rotation of device200 may be substantially vertical. Thus, for instance, by rotating LIDAR206 (and the attached camera 208), device 200 (and/or a computing systemthat operates device 200) can determine a three-dimensional map(including color information based on images from camera 208 anddistance information based on data from LIDAR 206) of a 360-degree viewof the environment of device 200. Additionally or alternatively, in someexamples, device 200 can be configured to tilt the axis of rotation ofrotating platform 210 (relative to stationary platform 214), therebysimultaneously adjusting the FOVs of LIDAR 206 and camera 208. Forinstance, rotating platform 210 may include a tilting platform thattilts in one or more directions to change an axis of rotation of device200.

In some examples, lens 224 can have an optical power to both collimate(and/or direct) emitted light beams 280 toward an environment of LIDAR206, and focus light 290 received from the environment onto a LIDARreceiver (not shown) of LIDAR 206. In one example, lens 224 has a focallength of approximately 120 mm. Other example focal lengths arepossible. By using the same lens 224 to perform both of these functions,instead of a transmit lens for collimating and a receive lens forfocusing, advantages with respect to size, cost, and/or complexity canbe provided. Alternatively however, LIDAR 206 may include separatetransmit and receive lenses. Thus, although not shown, LIDAR 206 canalternatively include a transmit lens that directs emitted light 280toward the environment, and a separate receive lens that focusesreflected light 290 for detection by LIDAR 206.

FIG. 3A shows another device 300 that includes co-aligned rotatingsensors, according to example embodiments. As shown, device 300 includesa rotating platform 310, a stationary platform 314, a housing 318, and alens 324 that are similar, respectively, to rotating platform 110,stationary platform 114, housing 118, and optical element(s) 124 (and/or130). Thus, device 300 may be similar to system 100 and/or device 200.Incident light 390 may include reflections of emitted light 380 (emittedby device 300), as well as external light (e.g., similar to light 292)originating from external light sources. Thus, unlike device 200, device300 is shown to include a single lens 324 for receiving incident light390 instead of a LIDAR lens (e.g., lens 224) and a physically separatecamera lens (e.g., lens 230).

FIG. 3B illustrates a cross-section view of device 300. As shown,housing 318 includes a transmitter 320, a receiver 322, lens 324, and anoptical element 354, which may be similar to, respectively, transmitter120, receiver 122, and optical element(s) 124 (and/or 130). Further, asshown, device 300 includes a shared space 340. For purposes ofillustration, FIG. 3B shows an x-y-z axis, in which the z-axis ispointing out of the page.

Transmitter 320 includes a plurality of light sources 342 a-c arrangedalong a curved focal surface 348. Light sources 342 a-c can beconfigured to emit, respectively, a plurality of light beams 344 a-chaving wavelengths within a wavelength range. For example, light sources342 a-c may comprise laser diodes that emit light beams 344 a-c havingthe wavelengths within the wavelength range. As shown, light beams 344a-c are reflected by mirror 350 through an exit aperture 352 into sharedspace 340 and toward lens 324.

To that end, light sources 342 a-c may include laser diodes, lightemitting diodes (LEDs), laser bars (e.g., diode bars), vertical cavitysurface emitting lasers (VCSELs), organic light emitting diodes (OLEDs),polymer light emitting diodes (PLEDs), light emitting polymers (LEPs),liquid crystal displays (LCDs), microelectromechanical systems (MEMS),or any other device configured to selectively transmit, reflect, and/oremit light beams 344 a-c. In some examples, light sources 342 a-c can beconfigured to emit light beams 344 a-c in a wavelength range that can bedetected by receiver 322. The wavelength range could, for example, be inthe ultraviolet, visible, and/or infrared portions of theelectromagnetic spectrum. In some examples, the wavelength range can bea narrow wavelength range, such as provided by lasers. In oneembodiment, the wavelength range includes a source wavelength of 905 nm.Additionally, light sources 342 a-c can optionally be configured to emitlight beams 344 a-c in the form of pulses. In some examples, lightsources 342 a-c can be disposed on one or more substrates (e.g., printedcircuit boards (PCB), flexible PCBs, etc.) and arranged to emit lightbeams 344 a-c toward mirror 350. To that end, mirror 350 may include anyreflective material suitable for reflecting the wavelengths of lightbeams 344 a-c through exit aperture 352 and toward lens 324.

Although FIG. 3B shows that curved focal surface 348 is curved in ahorizontal plane (e.g., x-y plane), additionally or alternatively, lightsources 344 a-c may be arranged along a focal surface that is curved ina vertical plane (e.g., perpendicular to the x-y plane, etc.). In oneexample, curved focal surface 348 can have a curvature in a verticalplane, and light sources 344 a-c can include additional light sourcesarranged vertically along focal surface 348 and configured to emit lightbeams directed toward mirror 350 and then reflected (by mirror 350)through exit aperture 352 and toward lens 324. In this example,detectors 362 a-c may also include additional detectors that correspondto the additional light sources. Further, in some examples, lightsources 342 a-c may include additional light sources arrangedhorizontally along curved focal surface 348. In one embodiment, lightsources 342 a-c may include sixty-four light sources that emit lighthaving a wavelength of 905 nm. For instance, the sixty-four lightsources may be arranged in four columns, each comprising sixteen lightsources. In this instance, detectors 362 a-c may include sixty-fourdetectors that are arranged similarly (e.g., four columns comprisingsixteen detectors each, etc.) along curved focal surface 368. However,in other embodiments, light sources 342 a-c and detectors 362 a-c mayinclude additional or fewer light sources and/or detectors than thoseshown.

Due to the arrangement of light sources 342 a-c along curved focalsurface 348, light beams 344 a-c, in some examples, may converge towardexit aperture 352. Thus, in these examples, exit aperture 352 could beminimally sized while being capable of accommodating the vertical andhorizontal extents of light beams 344 a-c. Additionally, in someexamples, a curvature of curved focal surface 348 can be based onoptical characteristics of lens 324. For example, curved focal surface348 may be shaped according to a focal surface of lens 324 (e.g., basedon shape and/or composition of lens 324) at transmitter 320.

As shown, light beams 344 a-c propagate in a transmit path that extendsthrough transmitter 320, exit aperture 352, and shared space 340 towardlens 324. Further, as shown, lens 324 may be configured to collimatelight beams 344 a-c into, respectively, light beams 346 a-c forpropagation toward an environment of device 300. In some examples,collimated light beams 346 a-c may then reflect off one or more objectsin the environment, and the reflections may then propagate back towarddevice 300 within incident light 390.

Further, as noted above, incident light 390 may also include externallight originating from one or more external light sources (e.g., solar,background light, street lamp, vehicle lamp, etc.), and at least aportion of the external light may also reflect off the one or moreobjects prior to propagating toward device 300. Incident light 390 maythen be focused by lens 324 into shared space 340 as focused light 348traveling along a receive path that extends through shared space 340toward optical element 354. For example, at least a portion of focusedlight 348 may be reflected on selectively (or partially) reflectivesurface 354 a as focused light 348 a-c toward receiver 322.

Thus, in some examples, lens 324 may be capable of both collimatingemitted light beams 344 a-c and focusing incident light 390 based onoptical characteristics (e.g., due to shape, composition, etc.) of lens324. In one particular embodiment, lens 324 can have an aspheric surface324 a outside of housing 318 (facing the environment of device 300) anda toroidal surface 324 b inside of housing 318 (facing shared space340). By using the same lens 324 to perform both of these functions,instead of a transmit lens for collimating and a receive lens forfocusing, advantages with respect to size, cost, and/or complexity canbe provided. However, in other embodiments, device 300 may alternativelyinclude a transmit lens for collimating emitted light beams 344 a-c, anda separate receive lens for focusing incident light 390. For instance,although not shown, device 300 may alternatively include a separatehousing (other than housing 318) that houses transmitter 320 such thatlight beams 344 a-c are emitted in a different space than shared space340. However, for the sake of example, device 300 is configured as shown(i.e., with a shared transmit/receive lens 324).

As shown, optical element 354 is located between transmitter 320 andshared space 340 along a path of focused light 348. Optical element 354may comprise any device having optical characteristics suitable forselectively or partially reflecting a first portion of focused light 348(incident on surface 354 a) toward receiver 322, and transmitting asecond portion of focused light 348 through optical element 354 andtoward image sensor 326.

In one example, optical element 354 (or a portion thereof) may comprisea material having partially reflective optical properties. For instance,optical element 354 may comprise a half mirror or beam splitter thatreflects a first portion of focused light 348 toward receiver 322, andtransmits a second portion of focused light 348 through optical element354 and toward image sensor 354.

In another example, optical element 354 (or a portion thereof) maycomprise a material having wavelength-based transmission and reflectionproperties. For instance, optical element 354 may comprise a dichroicmirror, which may be formed from a dielectric material, crystallinematerial, or any other material having wavelength-based reflectionand/or transmission properties. Thus, for instance, the dichroic mirrormay be configured to reflect wavelengths of light within the wavelengthrange of light beams 344 a-c (originated by transmitter 320 andreflected off one or more objects in the environment of device 300)toward receiver 322. Further, the dichroic mirror may be configured totransmit a portion of focused light 348 having wavelengths outside thewavelength range of light beams 344 a-c toward image sensor 326.

Thus, in some implementations, optical element 354 can be employed(e.g., as a light filter, etc.) to reduce an amount of external light(e.g., light originating from light sources other than light sources 342a-c) propagating toward receiver 322, by transmitting such externallight through optical element 354 toward image sensor 326. In otherimplementations, optical element 354 can be employed as a beam splitter(e.g., half mirror, etc.) that reflects a portion of focused light 348toward receiver 322 (without necessarily separating particularwavelengths from the reflected portion). Regardless of theimplementation, optical element 354 may be configured to allow bothreceiver 322 and image sensor 326 to detect respective portions ofincident light 390 (focused as focused light 348). As a result, forinstance, both receiver 322 and image sensor 326 may have co-alignedFOVs.

As shown, exit aperture 352 is included in optical element 354. In someexamples, optical element 354 can be formed from a transparent material(e.g., glass, etc.) that is coated with selectively or partiallyreflective material 354 a (e.g., dielectric material, crystallinematerial, other material having wavelength-based reflection and/ortransmission properties, partially reflective material, etc.). In afirst example, exit aperture 352 may correspond to a portion of opticalelement 354 that is not coated by the selectively or partiallyreflective material 354 a. In a second example, exit aperture 352 maycomprise a hole or cut-away in dichroic mirror 354.

However, in other examples, exit aperture 352 can be formed in adifferent manner. In one example, optical element 354 (e.g., beamsplitter, dichroic mirror, etc.) can be formed to have opticalcharacteristics that allow light wavelengths associated with light beams344 a-c to propagate from one side of optical element 354 (e.g., sidefacing transmitter 320) toward and out of an opposite side (e.g., sidefacing shared space 340), while preventing (or reflecting or reducing)an amount of such light from propagating in an opposite direction (e.g.,from surface 354 a toward opposite surface of element 354, etc.). Inanother example, optical element 354 may be configured to allowtransmission of light wavelengths associate with light beams 344 a-344 cand incident on optical element 354 at a particular angle, whilereflecting such light wavelengths incident on optical element 354 atdifferent angles. In this example, mirror 350 can be configured todirect light beams toward optical element 354 according to theparticular angle that allows transmission of light beams 344 a-c towardlens 324, and lens 324 can have optical characteristics such thatfocused light 348 propagates toward optical element 354 a at a differentangle than the particular angle.

Image sensor 326 may be similar to image sensor 126. For example, imagesensor 326 may include an array of sensing elements (e.g., CCDs, etc.)that are arranged to detect a first portion of the focused lighttransmitted through optical element 354. Further, image sensor 326 mayoutput an image including image pixel data (e.g., color information,etc.) indicative of an amount of light incident on the array of sensingelements during a given exposure time. Thus, the output image maycorrespond to an image of the scene that is simultaneously scanned bydevice 300 (using emitted light beams 344 a-344 c) based on detection ofexternal light (included in incident light 390 along with reflections oflight beams 344 a-344 c). To that end, the external light detected byimage sensor 326 may indicate additional information (e.g., RGB colorinformation, etc.) due to the larger extent of wavelengths in theexternal light within incident light 390 that may be detectable bysensing elements of image sensor 326.

Further, in line with the discussion above, a second portion of focusedlight 348 (incident on optical element 354) may be reflected at opticalelement 354 toward an entrance aperture 356 of receiver 322. In someexamples, entrance aperture 356 may comprise a filtering window or lightfilter configured to transmit wavelengths in the wavelength range oflight beams 344 a-c (e.g., source wavelength) originating from lightsources 342 a-c, while attenuating other wavelengths (e.g., externallight). Thus, as shown, at least a portion of focused light, shown aslight 348 a-c, propagates toward detectors 362 a-c.

Detectors 362 a-c may comprise photodiodes, avalanche photodiodes,phototransistors, charge coupled devices (CCD), CMOS sensors, thermopilearrays, cryogenic detectors, or any other sensor of light configured todetect focused light 348 a-c having wavelengths in the wavelength rangeof the emitted light beams 344 a-c (e.g., the source wavelength).

As shown, detectors 362 a-c can be arranged along curved focal surface368 of receiver 322. Although curved focal surface 368 is shown to becurved along the x-y plane (horizontal plane), additionally oralternatively, curved focal surface 368 can be curved in a verticalplane. Thus, similarly to focal surface 348, a curvature of focalsurface 368 may also be defined by optical characteristics of lens 324.For example, curved focal surface 368 may correspond to a focal surfaceof light projected by lens 324 at receiver 322.

Thus, for example, detector 362 a may be configured and arranged toreceive focused light portion 348 a that corresponds to a reflection ofcollimated light beam 346 a off one or more objects in the environmentof device 300. Further, as noted above, collimated light beam 346 acorresponds to light beam 344 a emitted by the light source 342 a. Thus,detector 362 a may be configured to receive light that was originallyemitted by light source 342 a based on the arrangement and opticalcharacteristics of the various components of device 300. Similarly,detector 362 b may be arranged to receive light that was originallyemitted by light source 342 b, and detector 362 c may be arranged toreceive light that was emitted by light source 342 c.

With this arrangement, device 300 (and/or a computing device thatoperates device 300) may determine at least one aspect of one or moreobjects (e.g., off which at least a portion of focused light 348 a-348 cwas reflected) by comparing detected characteristics of received light348 a-c (measured using detectors 362 a-c) with characteristics ofemitted light beams 344 a-c (transmitted using light sources 344 a-c).For example, by comparing emission times when light beams 344 a-c wereemitted by light sources 342 a-c and detection times when detectors 362a-c received focused light 348 a-c, a distance between device 300 andthe one or more objects could be determined. As another example,respective modulation characteristics (e.g., power level, waveformshape, etc.) of emitted light 344 a-c and detected light 348 a-c can becompared to determine information about the one or more objects (e.g.,distance, speed, material properties, etc.). Thus, in some examples,various characteristics of the one or more objects such as distance,speed, shape, material, texture, among other characteristics, couldoptionally be determined using data from receiver 322.

Further, due to the co-aligned relative arrangement of receiver 322 andimage sensor 326, image pixel data (e.g., color information, etc.)collected using image sensor 326 can be more suitable for efficientlymapping with LIDAR data collected using receiver 322. For example, acomputing device (e.g., controller 104, etc.) can operate image sensor326 to capture image pixel data according to emission times of lightbeams 344 a-c emitted by transmitter 320, and/or detection times oflight beams 348 a-c detected by receiver 322. Further, for example, theshared (co-aligned) receive path of focused light 348 (with portionsthereof respectively detected by receiver 322 and image sensor 326) canfurther improve mapping image pixel data from image sensor 326 to LIDARdata from receiver 322 due to the overlapping FOVs of both sensors 322and 326.

In some examples, device 300 may be rotated about an axis to determine athree-dimensional map of the surroundings of device 300. For example,device 300 may be rotated about a substantially vertical axis (extendingout of the page) as illustrated by arrow 394. Further, although device300 is shown to be rotated in a counterclockwise direction (e.g., arrow394), device 300 could alternatively be rotated in a clockwisedirection. Further, in some examples, device 300 may be rotated for 360degree (e.g., complete) rotations. In other examples however, device 300may be rotated back and forth along a portion of a 360 degree range ofpointing directions of device 300. For example, device 300 could bemounted on a platform that wobbles back and forth about an axis withoutmaking a complete rotation.

To that end, as shown, the arrangement of light sources 342 a-c anddetectors 362 a-c may define a particular vertical LIDAR FOV of device300. In one embodiment, the vertical LIDAR FOV is 20°. Further, in oneembodiment, complete rotations (360 degrees) of device 300 may define a360° horizontal LIDAR FOV of device 300. In this regard, a rate of therotation of device 303 may define a LIDAR refresh rate of device 300. Inone embodiment, the refresh rate is 10 Hz. Further, the LIDAR refreshrate together with the arrangement of light sources 342 a-c anddetectors 352 a-c may define a LIDAR angular resolution of device 300.In one example, the LIDAR angular resolution is 0.2°×0.3°. However, itis noted that the various parameters described above, including therefresh rate, the angular resolution, etc., may vary according tovarious configurations of device 300.

Further, in some examples, regardless of the specific parametersselected for the LIDAR scanning operations of device 300, the particularrelative arrangement of the various components shown in device 300 canfacilitate operating the image sensor 326 in a coordinated manner withreceiver 322. In a first example, if a light pulse rate of emitted lightbeams 344 a-c is adjusted, an image capture rate by image sensor 326 canbe similarly adjusted to facilitate time-domain based mapping of imagepixel data collected using image sensor 326 to LIDAR data collectedusing receiver 322. In a second example, device 300 may continue toprovide a suitable amount of overlapping data points (e.g., image pixelsthat correspond to LIDAR data points, etc.) between collected LIDAR dataand collected image data even as device 300 continues to rotate. Forinstance, the particular relative arrangement of transmitter 320,receiver 322, and image sensor 326 shown may remain substantiallyunchanged in response to rotation of device 300. Thus, with thisarrangement, the pointing directions of image sensor 326 and receiver322 may simultaneously and similarly change for any direction ofrotation (e.g., clockwise or counterclockwise) and/or rate of rotationof device 300. As a result, for instance, the LIDAR data (from receiver322) and the image data (from image sensor 326) may include a suitableamount of overlapping data points (e.g., both in terms of time of datacollection and FOVs associated with the collected data), even ifrotation characteristics of device 300 vary. Other examples are possibleas well.

It is noted that device 300 may include additional, fewer, or differentcomponents than those shown. Further, it is noted that the sizes andshapes of the various components 300 are illustrated as shown only forconvenience in description and are not necessarily to scale. Further, insome examples, one or more of the components shown can be combined,arranged, or separated in a variety of different configurations withoutdeparting from the scope of the present disclosure.

In a first example, light sources 342 a-c and/or detectors 362 a-c maybe configured differently (e.g., along different curved focal surfaces,along linear or flat focal surfaces, with different numbers of lightsources/detectors, etc.).

In a second example, transmitter 320 can be physically implemented as aseparate component outside housing 318. Thus, housing 318 couldalternatively include receiver 322 and image sensor 326, but nottransmitter 320.

In a third example, image sensor 326 can be alternatively positioned ina different location within housing 318. For instance, image sensor 318can be arranged adjacent to receiver 322. In this instance, a relativelysmall portion of focused light 348 may correspond to a detection area ofdetectors 362 a-c, and a remaining portion of focused light 348 can beintercepted by image sensor 326 at any location within shared space 340(with or without optical element 354).

In a fourth example, an orientation of image sensor 326 can be differentthan the orientation shown. For instance, image sensor 326 (and/orsensing elements thereof) could be aligned according to a focal surfacedefined by lens 324 (e.g., perpendicular to a focal plane of lens 324,along a curved focal surface, defined by lens 324 etc.). Alternativelyor additionally, for instance, image sensor 326 may include one or morecolumns of sensing elements (e.g., similar to sensing elements 128) thatare aligned (e.g., parallel, etc.) with an axis of rotation of device300.

In a fifth example, the locations of receiver 322 and image sensor 326can be interchanged. For instance, optical mirror 354 can bealternatively configured to reflect wavelengths of light outside thewavelength range of emitted light beams 344 a-c, and to transmit lightin the wavelength range through optical element 354. Thus, in thisinstance, receiver 322 can be alternatively positioned behind opticalelement 354 (e.g., in the location shown for image sensor 326), andimage sensor 326 can be alternatively positioned in the location shownfor receiver 322. Other examples are possible as well.

FIG. 4 illustrates a partial view of yet another device 400 thatincludes co-aligned rotating sensors, according to example embodiments.As shown, device 400 includes a transmitter 420, a receiver 422, animage sensor 426, and a lens 430, which may be similar, respectively, totransmitter 120, receiver 122, image sensor 126, and optical element(s)124 (or 126), for example. Further, as shown, device 400 also includes awaveguide 424 (e.g., optical waveguide, etc.) and an aperture 452.

In some implementations, one or more of the components of device 400shown can be used with device 300 instead of or in addition to thevarious components included in housing 318 (and shown in FIG. 3B). Thus,similarly to device 300, device 400 can be configured to rotate about anaxis while maintaining the particular relative arrangement oftransmitter 420, receiver 422, and image sensor 426 shown in FIG. 4.

Further, as shown, device 400 may measure incident light 490 scatteredby an object 498 within a scene in an environment of device 400. To thatend, incident light 490 may be similar to incident light 390 and mayinclude reflections of light originating from transmitter 420 as well asexternal light originating from one or more external light sources (andreflecting off object 498). Thus, in some examples, lens 430 may beconfigured as a shared lens that focuses light for detection by receiver422 and image sensor 426 according to a shared co-aligned receive path.

Transmitter 420 may include one or more light sources, similarly to anyof light sources 342 a-c. As shown, transmitter 420 emits light (e.g.,similar to any of light beams 344 a-c) toward a first side 424 a ofwaveguide 424.

Waveguide 424 may include any optical waveguide that guides propagationof light (e.g., via internal reflection at walls of waveguide 424,etc.). As shown, for instance, waveguide 424 may guide emitted lightfrom transmitter 420 (entering waveguide 424 at side 424 a) towardanother side of waveguide 424 (opposite to side 424 a). To that end, forexample, waveguide 424 can be formed from a glass substrate (e.g., glassplate, etc.) or any other material at least partially transparent to oneor more wavelengths of the light emitted by transmitter 420. In someexamples, as shown, waveguide 424 may be proximally positioned and/or incontact with image sensor 426 such that the guided light fromtransmitter 420 is transmitted through aperture 452 toward lens 430.

As shown, image sensor 426 includes an array of sensing elements,exemplified by sensing elements 428 a-d. Sensing elements 428 a-d may besimilar to sensing elements 128, and may include any type of device(e.g., CCD, etc.) that detects a portion of focused light 448 (focusedby lens 430 toward image sensor 426). Thus, in one example, data fromthe sensing elements of image sensor 426 can be used to generaterespective image pixels of an image (captured by image sensor 426) ofthe scene that includes object 498.

Further, as shown, aperture 452 may correspond to a region of imagesensor 426 that allows transmission of light between waveguide 424 andlens 430. For example, aperture 452 may correspond to a region of imagesensor 426 that does not include sensing elements (e.g., cavity, etc.).As another example, aperture 452 may correspond to a region that istransparent to wavelengths of light similar to wavelengths of theemitted light originating from transmitter 420. Other examples arepossible as well.

Thus, in an example scenario, lens 430 may be configured to direct theemitted light (originating from transmitter 420, entering waveguide 424at side 424 a, and transmitted through aperture 452 toward lens 430)toward object 498. In the scenario, at least a portion of the emittedlight may then reflect off object 498 back toward lens 430 as incidentlight 490. As noted above, incident light 490 may also include externallight originating from external light source(s) and reflecting offobject 498 together with the light originating from transmitter 420.Continuing with the scenario, lens 430 may then focus incident light490, as focused light 448, toward image sensor 426 and aperture 452.Waveguide 424 may then guide at least a portion of focused light 448(i.e., the portion transmitted through aperture 452 into waveguide 424),via internal reflection at walls of waveguide 424 for instance, insidewaveguide 424. For instance, where waveguide 424 is a rectangularwaveguide as shown, waveguide 424 can guide at least a portion offocused light 448 toward side 424 b.

Further, in the scenario, at least a portion of the guided lightincident on side 424 b may propagate out of waveguide 424 towardreceiver 422. For example, waveguide 424 can be configured as a leakywaveguide that allows photons to escape through side 424 b for detectionby receiver 422. As another example, although not shown, device 400 mayinclude a mirror (e.g., reflective material, half mirror, dichroicmirror, etc.) disposed within or otherwise coupled to waveguide 424 toreflect at least a portion of the guided light out of waveguide 424through side 424 b. Other examples are possible.

Thus, with this arrangement, receiver 422 may detect reflections of thelight emitted by transmitter 420 (and reflected off object 498), andimage sensor 426 can simultaneously or in another coordinated fashiondetect external light from the scene (reflecting off object 498) andfocused onto sensing elements (e.g., 428 a-d) of image sensor 426. Forexample, lens 430 may be configured to have an optical power forfocusing a first portion of incident light 490 onto a location ofaperture 452, where the first portion corresponds to the emitted lightoriginating at transmitter 420. Further, for example, lens 430 may beconfigured to have the optical power to focus a second portion ofincident light 490 (e.g., originating from external light source(s),etc.) toward a (e.g., larger) detection area where the sensing elementsof image sensor 426 are located. For instance, aperture 452 can belocated at a focal distance of lens 430. In this instance, a light beam(transmitted through aperture 452 toward lens 430) could diverge towardobject 498 based on optical characteristics of lens 430 and reflect backto lens 430. Further, due to the aperture being at the focal distance oflens 430, the reflections of the light beam may be focused back intoaperture 452, whereas the external light focused by lens 430 may befocused onto other regions of the focal plane of lens 430 (e.g., regionswhere sensing elements of image sensor 426 are located). Other examplesare possible.

It is noted that the sizes, positions, and shapes of the variouscomponents and features shown in FIG. 4 are not necessarily to scale,but are illustrated as shown for convenience in description. Further, insome embodiments, one or more of the components shown can be combined,or divided into separate components. Further, in some examples, system300 may include fewer or more components than those shown.

In a first example, device 400 can alternatively include multipleapertures that define optical paths through image sensor 426. Further,each aperture may be coupled to a waveguide, transmitter, and/orreceiver, similarly to aperture 452. By doing so for instance, multipleLIDAR data points can be captured simultaneously with the capture of animage using image sensor 426. In a second example, the distance betweenimage sensor 426 (and aperture 452) relative to lens 430 can vary. In athird example, the sensing elements in image sensor 426 can bealternatively implemented as separate physical structures. In a fourthexample, although the array of sensing elements in image sensor 426 isarranged along a linear (flat) surface, the sensing elements canalternatively be arranged along a different surface. For instance, agiven column of sensing elements (e.g., extending from elements 428 c to428 d) can be arranged at a different distance to lens 430 than anothercolumn of sensing elements in the image sensor 426. In a fifth example,receiver 422 can be implemented to alternatively or additionally overlapone or more other sides of waveguide 424). In a sixth example, imagesensor 426 may include a different number of sensing elements thanshown. Other examples are possible.

Additionally, it is noted that the sensor configurations of devices 200,300, 400 are only for the sake of example. Thus, other example sensorconfigurations are possible as well without departing from the scope ofthe present disclosure.

In a first example, various LIDAR device arrangements different thanthose shown can be combined with an image sensor on a rotating platformin a variety of different ways such that the image sensor and a receiverof the LIDAR remain in a particular relative arrangement in response torotation of the platform. For instance, a LIDAR device can be configuredto have a LIDAR transmitter implemented separately from a LIDARreceiver. In this instance, the LIDAR receiver can be coupled to theimage sensor to achieve a co-aligned receive path (e.g., particularrelative arrangement, co-aligned pointing directions, and/orsimultaneously changing respective pointing directions) for the LIDARreceiver and the image sensor. To that end, the particular relativearrangement of the LIDAR receiver and the image sensor on the rotatingplatform can vary depending on the particular configuration and opticalcharacteristics of a corresponding LIDAR system within an exampledevice.

In a second example, different types of sensors can be employed inaddition to or instead of a LIDAR and an image sensor. In oneimplementation, referring back to FIG. 2, LIDAR 206 can be replaced withRADAR, SONAR, or any other type of active sensor, and camera 208 can bereplaced by a microphone, a pressure sensor, an antenna, or any othertype of passive sensor. In another implementation, continuing with theexample of FIG. 2, LIDAR 206 and camera 208 can both be replaced withactive sensors (e.g., RADAR and SONAR, etc.), or can both be replacedwith passive sensors (e.g., microphone and pressure sensor, etc.). Othercombinations of sensors are possible as well.

III. EXAMPLE SCANNING METHODS AND COMPUTER READABLE MEDIA

As noted above, in some scenarios, capturing data using a rotatingsensor can result in image smearing (e.g., a light detector thatintercepts light over a long period of time while the sensor is rotatingmay detect light from several FOVs and/or regions of the scene).However, for instance, detectors (e.g., array of single photon avalanchediodes or SPADs) of a LIDAR receiver may be able to detect a returninglight pulse within a relatively short time window. Thus, a rotatingLIDAR may mitigate the effects of the rotation by transmitting shortpulses and listening for returning light pulses over a short time window(e.g., nanoseconds, microseconds, etc.) relative to the rate of rotationof the LIDAR.

On the other hand, a CCD element of an image sensor, for example, may bemore suitable for operation according to a relatively longer exposuretime that allows collecting a sufficient amount of light that indicatescolor information indicated by incident light (e.g., intensity of red,green, or blue light). However, in some instances, exposing the CCDelement for such exposure time may cause the CCD element to receivelight originating from multiple points in the scene rather than a singlepoint. Further, for instance, a shorter exposure time may beinsufficient to detect a sufficient amount of the incident light (e.g.,especially in low light conditions) to provide such light intensityinformation.

Thus, an example system may address these issues by adding a first pixel(captured at a first time while the sensor is rotating) to a secondpixel (captured at a second time while the sensor is rotating). Further,the example system may select which image pixels to add based on avariety of factors, such as a rate of rotation of the rotating device, adistance between an imaged object and the rotating device (e.g.,indicated by LIDAR data), and/or optical characteristics of the rotatingdevice (e.g., lens characteristics, position of image sensor relative tofocal length of lens, etc.).

FIG. 5 is a first conceptual illustration of images 502, 504 based ondata from one or more rotating sensors, according to exampleembodiments. In the scenario shown, image 502 may be based on imagesensor data obtained using any of image sensors 126, 326, 426, orcameras 108, 208, at a first time. Similarly, image 504 may be based onimage sensor obtained at a second time after the first time. Forinstance, images 502, 504 may be obtained at different times while anyof system 100, devices 200, 300, and/or 400 are rotating theirrespective image sensors (or cameras).

To that end, for example, each grid cell in the grid representation ofimage 502 may correspond to a particular image pixel (or a group ofimage pixels) in an image indicated by an array of sensing elements(e.g., sensing elements 128 or 428) at a first time. For instance, gridcell/image pixel 502 a may be determined based on image pixel data froma first sensing element (or group of sensing elements) in array 428,image pixel 502 b may be determined based on data from a second sensingelement (or group) adjacent to the first sensing element in a horizontaldirection (e.g., same row), and image pixel 502 c may be determinedbased on data from a third sensing element (or group) adjacent to thesecond sensing element in the horizontal direction. By way of example,image pixels 502 a, 502 b, 502 c, may be determined using data from,respectively, sensing elements 428 a, 428 b, 428 c.

It is noted that the numbers and sizes of image pixels 502 a-c in image502 are not necessarily to scale but are only for convenience indescription. In practice, for instance, image 502 may include hundredsor thousands or any other number of image pixels.

Continuing with the scenario of FIG. 5, image pixels 504 a, 504 b, 504 cof image 504 may be based on data from the same respective sensingelements used to generate image pixels 502 a, 502 b, 502 c of image 502but at the second time after the first time when the data of image 502was captured.

Further, in the scenario shown, images 502, 504 may indicate astar-shaped object in the scene scanned by the rotating sensor. As shownin image 502 for instance, the imaged object is indicated within a firstimage pixel column that includes pixel 502 a and a second image pixelcolumn that includes pixel 502 b. As shown image 504, the imaged objectappears to have moved to the right side of the image (e.g., due torotation of the image sensor) such that it is indicated within the twoimage pixel columns that include pixels 504 b and 504 c.

Thus, in some implementations, an example system may combine (e.g., add,sum, etc.) data of the first image pixel column (i.e., which includespixel 502 a) of image 502 to data of a corresponding image pixel column(i.e., which includes pixel 504 b) of image 504. Alternatively oradditionally, the example system can combine other corresponding imagepixel columns from the two images to (e.g., column of pixel 502 b withcolumn of pixel 504 c). Thus, the example system can generate acomposite image that simulates a longer exposure time (e.g., sum ofexposure times used in the two images), thereby achieving an improvedimage quality of the composite image relative to the quality associatedwith just one of images 502 or 504. Further, in line with the discussionabove, the example system can optionally synchronize the capture time ofone or both of images 502, 504 with emission or detection times of lightpulses associated with a co-aligned LIDAR (e.g., LIDARs 106, 206, 306,406, etc.). By doing so, for instance, the example system can improvesensor fusion processes for mapping LIDAR data with image sensor data(e.g., one or more image pixels associated with image 502 or 504 can bematched to data points in the LIDAR data that were collected at asimilar or same time, etc.).

It is noted that the scenario described above for the image sensoroperation is only for the sake of example. Other examples are possible.For example, the image sensor may alternatively capture just one columnof image pixels (e.g., using a column of sensing elements, etc.) at thefirst time associated with image 502 rather than the entire image, andthe image sensor may alternatively capture just one image pixel columnat the second time associated with image 504 rather than the entireimage. Other variations are possible as well.

FIG. 6 is a second conceptual illustration of images 602, 604 based ondata from one or more rotating sensors, according to exampleembodiments. In the scenario of FIG. 6, images 602, 604 may bedetermined, respectively, in a similar manner to the determination ofimages 502, 504 (e.g., at different times using data from a rotatingimage sensor, etc.). To that end, image pixels 602 a, 602 b, 602 c, 604a, 604 b, 604 c, may be similar (e.g., based on data from the samerespective sensing elements), respectively, to image pixels 502 a, 502b, 502 c, 504 a, 504 b, 504 c.

As noted above, the choice of which image pixel to add from image 502 toa corresponding image pixel from image 504 may be based at least in parton rotation characteristics of a rotating platform supporting the imagesensor that captured images 502, 504.

By way of example, the scenario shown in FIG. 6 illustrates a scenariowhere the image sensor is rotating at a higher rate of rotation than therate of rotation of the image sensor in the scenario of FIG. 5. Thus, asshown in FIG. 6, the star-shaped imaged object appears to have movedfurther toward a right side of image 604 than the corresponding imagedobject shown in image 504. As such, unlike the scenario in FIG. 5, anexample system in this scenario may generate an image pixel column in acomposite image of the scene based on a sum of a column of pixels inimage 602 (including pixel 602 a) and a corresponding column of pixelsin image 604 (including pixel 604 c) that is relatively closer to theright side of image 604.

In particular, although the scenarios of FIGS. 5 and 6 indicate that animaged object appears to move in a single linear direction due to therotation of the associated image sensor, in some scenarios, the rotationof the image sensor may alternatively cause various other distortions tothe appearance of the imaged object. By way of example, the changing FOVof the image sensor (e.g., change of pointing direction and/or viewpointof the image sensor, etc.) may alternatively cause the imaged object toappear rotated, re-sized, etc., in image 604 relative to an appearanceof the imaged object in image 602. As another example, referring back toFIG. 2, a tilting direction of rotating platform 210 relative tostationary platform 214 may cause an imaged object captured by camera208 in a plurality of images (as camera 208 rotates) may appear to bemoving in a vertical direction. As yet another example, referring backto FIGS. 3A-3B, the optical characteristics of lens 324 (e.g., surfaceshapes of surfaces 324 a, 324 b, etc.) may cause the shape of the imagedobject to become distorted as device 300 rotates according to arrow 394.Other examples are possible as well depending on the particularconfiguration of the device that includes the co-aligned rotatingsensors.

FIG. 7 is a third conceptual illustration of images 702, 704, 706 thatare based on data from one or more rotating sensors, according toexample embodiments. As shown, images 702, 704 may correspond to gridrepresentations of images captured using a rotating sensor at differenttimes similarly to, respectively, images 602, 604 for example. Further,as shown, image 706 may correspond to a composite image generated basedon a sum of image pixel data from at least a first pixel 702 a in image702 and a second pixel 704 a from image 704. In the scenario of FIG. 7,an example system may capture pixel 702 a (shown in image 702) using afirst sensing element (or group of sensing elements) of a rotating imagesensor at a first time, and pixel 704 a using a second different sensingelement (or group) of the rotating image sensor at a second time.

As noted above, in some implementations, an example system may beconfigured to select which image pixel(s) to add to pixel 702 b fromimage 704 based on a variety of factors in addition to or instead of therate of rotation of the rotating sensor.

In a first example, the example system may compute an expected pixellocation of a point in the imaged object (indicated by pixel 702 a) atthe second time associated with image 704 based on the rate of rotationof the rotating sensor as well as a configuration of the device thatsupports the rotating sensor (e.g., tilting direction of rotatingplatform 210 relative to stationary platform 214, opticalcharacteristics of lens 324, position of image sensor 426 relative tolens 430, etc.). The example system may then operate, at the secondtime, the particular sensing element (or group of sensing elements) tocapture the image pixel data (e.g., pixel 704 a) identified based on thecomputed expected pixel location.

In a second example, the example system may compute the expected pixellocation based on a distance between the portion of the objectassociated with pixel 702 a and the rotating sensor(s). For instance, afurther object from the image sensor may appear to move less than acloser object after a same amount of rotation of the image sensor. Tothat end, for instance, the example system can process LIDAR datadetected by a LIDAR receiver (e.g., receiver 122) at or within athreshold to the first time when image pixel 702 a (and/or image 702)was captured to estimate a distance between the image sensor (or theLIDAR) and the imaged object. The system can then use the estimateddistance (as well as an amount of rotation or change in pointingdirection(s) of the rotating sensor(s) between the first time and thesecond time) as a basis to compute an expected apparent motion of theobject (or portion thereof) associated with pixel 702 a. In turn, theexample system may identify the sensing elements of the image sensorassociated with image pixel 704 a as corresponding, at the second time,to the same object (or portion thereof) indicated by pixel 702 a at thefirst time.

In a third example, the example system may employ a sensor fusionalgorithm (or a computer program product storing an algorithm)configured to accept data from the image sensor and/or the LIDAR sensoras an input. The data may include, for example, data from LIDAR 106and/or camera 108 (e.g., images 702 and/or 704) or any other datarepresenting information sensed by sensors during rotation of the imagesensor (and/or LIDAR sensor). The sensor fusion algorithm may include,for example, a Kalman filter, a Bayesian network, an algorithm for someof the functions of the methods herein, or any other algorithm. Forinstance, the algorithm may identify features/edges of objects in thescanned scene and then match those identified features with features inimage 704 to determine that pixel 704 a corresponds to the object (orportion thereof) indicated by pixel 702 a.

Thus, regardless of the implementation, an example system can combinedata from pixel 702 a with data from pixel 704 a (captured at adifferent time) to generate pixel 706 a of composite image 706, therebysimulating an image pixel captured over a longer exposure time (e.g.,sum of exposure times associated with pixels 702 a and 704 a).

Further, in some implementations, additional pixels captured atsubsequent times (and corresponding to the same object or portionthereof indicated by pixel 702 a) can be added to pixels 702 a and 704 ato achieve a further improved image pixel 706 a of composite image 706.For instance, an example system may compute multiple locations of theobject over time based on the variety of factors described above (e.g.,rotation characteristics, LIDAR distance information, opticalcharacteristics of lenses, etc.). In turn, the example system can obtaintwo, three, or more image pixels (captured at different times) accordingto the computed path of the apparent motion of the object, as long asthe object is expected to remains within the FOV of the imaging sensor(e.g., imaged at any of the image pixels of the plurality of imagesobtained while the image sensor is rotating).

FIG. 8 is a flowchart of a method 800, according to example embodiments.Method 800 presents an embodiment of a method that could be used withany of system 100, devices 200, 300, and/or 400, for example. Method 800may include one or more operations, functions, or actions as illustratedby one or more of blocks 802-806. Although the blocks are illustrated ina sequential order, these blocks may in some instances be performed inparallel, and/or in a different order than those described herein. Also,the various blocks may be combined into fewer blocks, divided intoadditional blocks, and/or removed based upon the desired implementation.

In addition, for method 800 and other processes and methods disclosedherein, the flowchart shows functionality and operation of one possibleimplementation of present embodiments. In this regard, each block mayrepresent a module, a segment, a portion of a manufacturing or operationprocess, or a portion of program code, which includes one or moreinstructions executable by a processor for implementing specific logicalfunctions or steps in the process. The program code may be stored on anytype of computer readable medium, for example, such as a storage deviceincluding a disk or hard drive. The computer readable medium may includea non-transitory computer readable medium, for example, such ascomputer-readable media that stores data for short periods of time likeregister memory, processor cache and Random Access Memory (RAM). Thecomputer readable medium may also include non-transitory media, such assecondary or persistent long term storage, like read only memory (ROM),optical or magnetic disks, compact-disc read only memory (CD-ROM), forexample. The computer readable media may also be any other volatile ornon-volatile storage systems. The computer readable medium may beconsidered a computer readable storage medium, for example, or atangible storage device. In addition, for method 800 and other processesand methods disclosed herein, each block in FIG. 8 may representcircuitry that is wired to perform the specific logical functions in theprocess.

At block 802, method 800 involves scanning a scene using a lightdetection and ranging (LIDAR) sensor that emits light pulses toward thescene and detects reflections of the emitted light pulses from thescene. For example, LIDAR 206 may be configured to emit light 280 (e.g.,in the form of a sequence of pulses) toward the scene, and detectreflected light 290 (e.g., in the form of reflected light pulses, etc.).

At block 804, method 800 involves generating an image of the scene usingan image sensor that detects external light originating from one or moreexternal light sources. For example, lens 430 may focus incident light490 (e.g., which may include external light from external sources suchas the sun, street lamps, etc.) onto image sensor 426 to generate animage of object 498 (off which incident light 490 is reflected towarddevice 400).

At block 806, method 800 involves rotating a platform that supports theLIDAR sensor and the image sensor in a particular relative arrangement.For example, controller 104 may operate actuator 112 to rotate rotatingplatform 110 about an axis. Further, rotating platform may support LIDAR106 and camera 108 in a particular relative arrangement, and may thussimultaneously change the respective pointing directions of LIDAR 106and camera 108. Referring back to FIG. 2 by way of example, LIDAR 206and camera 208 may be connected to one another such that lenses 224 and230 have similar (or same) pointing directions, and then rotatingplatform 210 may simultaneously rotate both LIDAR 206 and camera 208 tosimultaneously change their respective pointing directions. As anotherexample, housing 318 of device 300 shown in FIG. 3B mounts LIDARtransmitter 320, LIDAR receiver 322, and image sensor 326 in aparticular relative arrangement relative to one another. Thus, as shownin FIG. 3A, rotating platform 310 may rotate housing 318 such thattransmitter 320, receiver 322, and image sensor 326 remain in theparticular relative arrangement in response to the rotation.

In some implementations, method 800 may involve associating data from aLIDAR receiver (of the LIDAR sensor) with data from the image sensor. Inone example, controller 104 can map image pixels indicated by the datafrom the image sensor to corresponding points in a data cloud indicatedby the LIDAR sensor. For instance, controller 104 can cause camera 108to capture one or more image pixels at particular respective times thatare based on emission and/or detection times of the LIDAR light pulses.

Further, in some implementations, method 800 may involve determining athree-dimensional (3D) representation of the scene based on data fromthe LIDAR receiver and data from the image sensor. For example, system100 can operate display 140 to render a display of a data point cloudsuch as an arrangement of points spaced apart based on detected lightpulses from the LIDAR sensor.

Further, in some implementations, method 800 may also involvedetermining a representation of the scene based on distance informationindicated by the LIDAR sensor and color information indicated by theimage sensor. Continuing with the example above, system 100 can assigncolors to one or more points in the point cloud by identifyingcorresponding image pixels in the data from the image sensor (e.g.,based on time of capture of the image pixels, overlapping FOVs of LIDAR106 and camera 108, feature or edge matching between the LIDAR data andthe image sensor data, etc.). System 100 can then update the data pointcloud (3D representation) rendered via display 140 to show colored datapoints according the colors determined using data from the image sensor.

In some implementations, method 800 may involve causing the image sensorto capture a plurality of images while the actuator is rotating theplatform. For example, controller 104 can operate camera 106 to capturea sequence of images (e.g., image 502, 504, etc.) while rotatingplatform 110 (which supports image sensor 126) is rotating.

In these implementations, method 800 may also involve determining a timedelay between capture of two consecutive images, and causing the imagesensor to capture the plurality of images according to the determinedtime delay. Referring back to FIG. 5 for example, an example system maydetermine that pixel locations of imaged star-shaped object in image 502correspond to pixel locations would be shifted by one column of imagepixels in image 504. Thus, for instance, the example system canoptionally capture image 502 as a first image at a first time, and image504 as a second consecutive image at a second time (e.g., withoutcapturing or storing intermediate images associated with a relativelysmall apparent motion of the imaged object corresponding to less than adistance of one pixel, etc.).

Alternatively or additionally, in these implementations, method 800 mayalso involve causing the image sensor to capture a given image of theplurality of images at a given time that is based on an emission time(or detection time) of one or more of the emitted light pulses. As such,an example system may improve alignment of LIDAR data from the LIDARsensor (at least in the time domain) with corresponding image sensordata from the image sensor. As a result, the example system can improvean accuracy of mapping color information from the image sensor todistance information from the LIDAR sensor.

In some implementations, method 800 involves determining a time delaybased on a rate of rotation of the platform (that supports the LIDARsensor and the image sensor in the particular relative arrangement),obtaining a first image pixel captured using a first sensing element inthe image sensor, obtaining a second image pixel captured using a secondsensing element in the image sensor after passage of the determined timedelay from a time when the first image pixel is captured, anddetermining a particular image pixel of the generated image (at block804) of the scene based on at least a sum of the first image pixel andthe second image pixel.

Referring back to FIG. 7 for example, an example system can use theco-aligned (at least in time domain) LIDAR data associated with imagepixel 702 a to select and/or identify an expected location, afterpassage of the determined time delay, of the corresponding image pixel704 a (e.g., due to the rotation at block 806). Then, after passage ofthe determined time delay from the time when image pixel 702 a wascaptured, the example system can obtain data indicative of image pixel704 a from the appropriate sensing elements that correspond to thelocation of image pixel 704 a. Thus, in these implementations, method800 may also involve selecting the second sensing element (e.g., sensingelement that corresponds to pixel 704 a) of the image sensor based on atleast distance information associated with the first image pixel andindicated by data from the LIDAR sensor.

IV. EXAMPLE VEHICLES

Some example implementations herein involve a vehicle that includes asensor system or device, such as system 100, devices 200, 300, and/or400 for instance. However, an example sensor disclosed herein can alsobe used for various other purposes and may be incorporated on orotherwise connected to any feasible system or arrangement. For instance,an example sensor system can be used in an assembly line setting tomonitor objects (e.g., products) being manufactured in the assemblyline. Other examples are possible as well.

Although illustrative embodiments herein include sensor(s) (e.g.,co-aligned rotating LIDAR and camera, etc.) mounted on a car, an examplesensor arrangement may additionally or alternatively be used on any typeof vehicle, including conventional automobiles as well as automobileshaving an autonomous or semi-autonomous mode of operation. Thus, theterm “vehicle” is to be broadly construed to cover any moving object,including, for instance, an airplane, other flying vehicle, a boat, atruck, a van, a semi-trailer truck, a motorcycle, a golf cart, anoff-road vehicle, a warehouse transport vehicle, or a farm vehicle, aswell as a carrier that rides on a track such as a rollercoaster,trolley, tram, or train car, among other examples.

FIG. 9 is a simplified block diagram of a vehicle 900, according to anexample embodiment. As shown, vehicle 900 includes a propulsion system902, a sensor system 904, a control system 906, peripherals 908, and acomputer system 910. In some embodiments, vehicle 900 may include more,fewer, or different systems, and each system may include more, fewer, ordifferent components. Additionally, the systems and components shown maybe combined or divided in any number of ways. For instance, controlsystem 006 and computer system 910 may be combined into a single system.

Propulsion system 902 may be configured to provide powered motion forthe vehicle 900. To that end, as shown, propulsion system 902 includesan engine/motor 918, an energy source 920, a transmission 922, andwheels/tires 924.

Engine/motor 918 may be or include any combination of an internalcombustion engine, an electric motor, a steam engine, and a Sterlingengine. Other motors and engines are possible as well. In someembodiments, propulsion system 902 may include multiple types of enginesand/or motors. For instance, a gas-electric hybrid car may include agasoline engine and an electric motor. Other examples are possible.

Energy source 920 may be a source of energy that powers the engine/motor918 in full or in part. That is, engine/motor 918 may be configured toconvert energy source 920 into mechanical energy. Examples of energysources 920 include gasoline, diesel, propane, other compressedgas-based fuels, ethanol, solar panels, batteries, and other sources ofelectrical power. Energy source(s) 920 may additionally or alternativelyinclude any combination of fuel tanks, batteries, capacitors, and/orflywheels. In some embodiments, energy source 920 may provide energy forother systems of vehicle 900 as well. To that end, energy source 920 mayadditionally or alternatively include, for example, a rechargeablelithium-ion or lead-acid battery. In some embodiments, energy source 920may include one or more banks of batteries configured to provide theelectrical power to the various components of vehicle 900.

Transmission 922 may be configured to transmit mechanical power fromengine/motor 918 to wheels/tires 924. To that end, transmission 922 mayinclude a gearbox, clutch, differential, drive shafts, and/or otherelements. In embodiments where the transmission 922 includes driveshafts, the drive shafts may include one or more axles that areconfigured to be coupled to the wheels/tires 924.

Wheels/tires 924 of vehicle 900 may be configured in various formats,including a unicycle, bicycle/motorcycle, tricycle, or car/truckfour-wheel format. Other wheel/tire formats are possible as well, suchas those including six or more wheels. In any case, wheels/tires 924 maybe configured to rotate differentially with respect to otherwheels/tires 924. In some embodiments, wheels/tires 924 may include atleast one wheel that is fixedly attached to the transmission 922 and atleast one tire coupled to a rim of the wheel that could make contactwith the driving surface. Wheels/tires 924 may include any combinationof metal and rubber, or combination of other materials. Propulsionsystem 902 may additionally or alternatively include components otherthan those shown.

Sensor system 904 may include a number of sensors configured to senseinformation about an environment in which vehicle 900 is located, aswell as one or more actuators 936 configured to modify a position and/ororientation of the sensors. As shown, sensor system 904 includes aGlobal Positioning System (GPS) 926, an inertial measurement unit (IMU)928, a RADAR unit 930, a laser rangefinder and/or LIDAR unit 932, and acamera 934. Sensor system 904 may include additional sensors as well,including, for example, sensors that monitor internal systems of vehicle900 (e.g., an O₂ monitor, a fuel gauge, an engine oil temperature,etc.). Other sensors are possible as well.

GPS 926 may be any sensor (e.g., location sensor) configured to estimatea geographic location of vehicle 900. To this end, the GPS 926 mayinclude a transceiver configured to estimate a position of vehicle 900with respect to the Earth.

IMU 928 may be any combination of sensors configured to sense positionand orientation changes of vehicle 900 based on inertial acceleration.The combination of sensors may include, for example, accelerometers,gyroscopes, compasses, etc.

RADAR unit 930 may be any sensor configured to sense objects in theenvironment in which vehicle 900 is located using radio signals. In someembodiments, in addition to sensing the objects, RADAR unit 930 mayadditionally be configured to sense the speed and/or heading of theobjects.

Similarly, laser range finder or LIDAR unit 932 may be any sensorconfigured to sense objects in the environment in which vehicle 900 islocated using lasers. For example, LIDAR unit 932 may include one ormore LIDAR devices, at least some of which may take the form of theLIDARS described in system 100, devices 200, 300, and/or 400 among otherLIDAR device configurations.

Camera 934 may be any camera (e.g., a still camera, a video camera,etc.) configured to capture images of the environment in which vehicle900 is located. To that end, camera 934 may take any of the formsdescribed above. For example, camera 934 may include an image sensor andmay take the form of any of the image sensors/cameras describe dinsystem 100, devices 200, 300, and/or 400 among other cameraconfigurations.

Control system 906 may be configured to control one or more operationsof vehicle 900 and/or components thereof. To that end, control system906 may include a steering unit 938, a throttle 940, a brake unit 942, asensor fusion algorithm 944, a computer vision system 946, navigation orpathing system 948, and an obstacle avoidance system 950.

Steering unit 938 may be any combination of mechanisms configured toadjust the heading of vehicle 900. Throttle 940 may be any combinationof mechanisms configured to control engine/motor 918 and, in turn, thespeed of vehicle 900. Brake unit 942 may be any combination ofmechanisms configured to decelerate vehicle 900. For example, brake unit942 may use friction to slow wheels/tires 924. As another example, brakeunit 942 may convert kinetic energy of wheels/tires 924 to an electriccurrent.

Sensor fusion algorithm 944 may be an algorithm (or a computer programproduct storing an algorithm) configured to accept data from sensorsystem 904 as an input. The data may include, for example, datarepresenting information sensed by sensor system 904. Sensor fusionalgorithm 944 may include, for example, a Kalman filter, a Bayesiannetwork, a machine learning algorithm, an algorithm for some of thefunctions of the methods herein (e.g., method 800, etc.), or any othersensor fusion algorithm. Sensor fusion algorithm 944 may further beconfigured to provide various assessments based on the data from sensorsystem 904, including, for example, evaluations of individual objectsand/or features in the environment in which vehicle 900 is located,evaluations of particular situations, and/or evaluations of possibleimpacts based on particular situations. Other assessments are possibleas well.

Computer vision system 946 may be any system configured to process andanalyze images captured by camera 934 in order to identify objectsand/or features in the environment in which vehicle 900 is located,including, for example, traffic signals and obstacles. To that end,computer vision system 946 may use an object recognition algorithm, aStructure from Motion (SFM) algorithm, video tracking, or other computervision techniques. In some embodiments, computer vision system 946 mayadditionally be configured to map the environment, track objects,estimate the speed of objects, etc.

Navigation and pathing system 948 may be any system configured todetermine a driving path for vehicle 900. Navigation and pathing system948 may additionally be configured to update a driving path of vehicle900 dynamically while vehicle 900 is in operation. In some embodiments,navigation and pathing system 948 may be configured to incorporate datafrom sensor fusion algorithm 944, GPS 926, LIDAR unit 932, and/or one ormore predetermined maps so as to determine a driving path for vehicle900.

Obstacle avoidance system 950 may be any system configured to identify,evaluate, and avoid or otherwise negotiate obstacles in the environmentin which vehicle 900 is located. Control system 906 may additionally oralternatively include components other than those shown.

Peripherals 908 may be configured to allow vehicle 300 to interact withexternal sensors, other vehicles, external computing devices, and/or auser. To that end, peripherals 908 may include, for example, a wirelesscommunication system 952, a touchscreen 954, a microphone 956, and/or aspeaker 958.

Wireless communication system 952 may be any system configured towirelessly couple to one or more other vehicles, sensors, or otherentities, either directly or via a communication network. To that end,wireless communication system 952 may include an antenna and a chipsetfor communicating with the other vehicles, sensors, servers, or otherentities either directly or via a communication network. The chipset orwireless communication system 952 in general may be arranged tocommunicate according to one or more types of wireless communication(e.g., protocols) such as Bluetooth, communication protocols describedin IEEE 802.11 (including any IEEE 802.11 revisions), cellulartechnology (such as GSM, CDMA, UMTS, EV-DO, WiMAX, or LTE), Zigbee,dedicated short range communications (DSRC), and radio frequencyidentification (RFID) communications, among other possibilities.

Touchscreen 954 may be used by a user to input commands to vehicle 900.To that end, touchscreen 954 may be configured to sense at least one ofa position and a movement of a user's finger via capacitive sensing,resistance sensing, or a surface acoustic wave process, among otherpossibilities. Touchscreen 954 may be capable of sensing finger movementin a direction parallel or planar to the touchscreen surface, in adirection normal to the touchscreen surface, or both, and may also becapable of sensing a level of pressure applied to the touchscreensurface. Touchscreen 954 may be formed of one or more translucent ortransparent insulating layers and one or more translucent or transparentconducting layers. Touchscreen 954 may take other forms as well.

Microphone 956 may be configured to receive audio (e.g., a voice commandor other audio input) from a user of vehicle 900. Similarly, speakers358 may be configured to output audio to the user.

Computer system 910 may be configured to transmit data to, receive datafrom, interact with, and/or control one or more of propulsion system902, sensor system 904, control system 906, and peripherals 908. To thisend, computer system 910 may be communicatively linked to one or more ofpropulsion system 902, sensor system 904, control system 906, andperipherals 908 by a system bus, network, and/or other connectionmechanism (not shown).

In a first example, computer system 910 may be configured to controloperation of transmission 922 to improve fuel efficiency. In a secondexample, computer system 910 may be configured to cause camera 934 tocapture images of the environment. In a third example, computer system910 may be configured to store and execute instructions corresponding tosensor fusion algorithm 944. In a fourth example, computer system 910may be configured to store and execute instructions for determining a 3Drepresentation of the environment around vehicle 900 using LIDAR unit932. In a fifth example, computer system 910 may be configured tooperate two co-aligned rotating sensors (e.g., LIDAR 932 and camera 934,etc.) synchronously and combine information from the two sensors (e.g.,color information from camera 934 and depth information from LIDAR 932),in line the discussion above for controller 104 of system 100. Otherexamples are possible as well.

As shown, computer system 910 includes processor 912 and data storage914. Processor 912 may comprise one or more general-purpose processorsand/or one or more special-purpose processors. To the extent thatprocessor 912 includes more than one processor, such processors couldwork separately or in combination.

Data storage 914, in turn, may comprise one or more volatile and/or oneor more non-volatile storage components, such as optical, magnetic,and/or organic storage, and data storage 914 may be integrated in wholeor in part with processor 912. In some embodiments, data storage 914 maycontain instructions 916 (e.g., program logic) executable by processor912 to cause vehicle 900 and/or components thereof to perform thevarious operations described herein. Data storage 914 may containadditional instructions as well, including instructions to transmit datato, receive data from, interact with, and/or control one or more ofpropulsion system 902, sensor system 904, control system 906, and/orperipherals 908.

In some embodiments, vehicle 900 may include one or more elements inaddition to or instead of those shown. In one example, vehicle 900 mayinclude one or more additional interfaces and/or power supplies. Inanother example, vehicle 900 may include system 100, devices 200, 300,and/or 400 in addition to or instead of one or more of the componentsshown. In such embodiments, data storage 914 may also includeinstructions executable by processor 912 to control and/or communicatewith the additional components. Still further, while each of thecomponents and systems are shown to be integrated in vehicle 900, insome embodiments, one or more components or systems may be removablymounted on or otherwise connected (mechanically or electrically) tovehicle 900 using wired or wireless connections. Vehicle 900 may takeother forms as well.

FIGS. 10A-10B collectively illustrate a vehicle 1000 equipped withsensor system 1004, according to example embodiments. Vehicle 1000 maybe similar to vehicle 900, for example. Although vehicle 1000 isillustrated as a car, as noted above, other types of vehicles arepossible. Furthermore, although vehicle 1000 may be configured tooperate in autonomous mode, the embodiments described herein are alsoapplicable to vehicles that are not configured to operate autonomously.

FIG. 10A shows a Right Side View, Front View, Back View, and Top View ofvehicle 1000. As shown, vehicle 1000 includes a sensor system 1010mounted on a top side of vehicle 1000 opposite a bottom side on whichwheels of vehicle 1000, exemplified by wheel 1024, are located. Wheel1024 may be similar to wheel(s) 924, for example.

Sensor system 1010 may be similar to or include system 100, devices 200,300, 400, and/or sensor system 904. For example, sensor system 1010 mayinclude an arrangement of co-aligned rotating sensors, such as LIDAR 106and camera 108 of system 100 for instance. Although sensor system 1010is shown and described as being positioned on a top side of vehicle1000, system 1010 could be alternatively positioned on any other part ofvehicle 1000, including any other side of vehicle 1000 for instance.

FIG. 10B shows that sensor system 1010 may be configured to scan anenvironment around vehicle 1000 by rotating about vertical axis 1012.For example, system 1010 may include a LIDAR (e.g., LIDAR 106, etc.)that is rotated while emitting one or more light pulses and detectingreflected light pulses off objects in an environment of vehicle 1000,for example. Further, for example, system 1010 may include a camera(e.g., camera 108, etc.) or other image sensor that is rotated with theLIDAR, in line with the discussion above.

Thus, as shown, a LIDAR of system 1010 may emit light 1080 (e.g.,similarly to emitted light 380, etc.) in a pointing direction of system1010, which is shown as a pointing direction toward a right side of thepage for example. With this arrangement, the LIDAR of system 1010 canemit light 1080 toward regions of the environment that are relativelyclose to the vehicle (e.g., a lane marker) as well as toward regions ofthe environment that are further away from the vehicle (e.g., a roadsign ahead of the vehicle). The LIDAR of system 1010 may also include areceiver that detects reflections of the emitted light (e.g., includedin incident light 1090). Further, system 1010 may include a co-alignedcamera that receives a portion of incident light 1090 (e.g., externallight, etc.) from a same or at least partially overlapping FOV as theLIDAR.

Further, vehicle 1000 can rotate system 1010 (e.g., the co-aligned LIDARand camera arrangement) about axis 1012 to change the pointingdirections of the sensors in system 1010 (e.g., LIDAR and camera)simultaneously. In one example, vehicle 1000 may rotate system 1010about axis 1012 repeatedly for complete rotations. In this example, foreach complete rotation of system 1010 (or one or more componentsthereof), system 1010 can scan a 360° FOV around vehicle 1000. Inanother example, vehicle 1000 may rotate system 1010 about axis 1012 forless than a complete rotation (e.g., to scan a limited horizontal FOVrather than a complete 360° FOV).

V. CONCLUSION

The particular arrangements shown in the Figures should not be viewed aslimiting. It should be understood that other implementations may includemore or less of each element shown in a given Figure. Further, some ofthe illustrated elements may be combined or omitted. Yet further, anexemplary implementation may include elements that are not illustratedin the Figures. Additionally, while various aspects and implementationshave been disclosed herein, other aspects and implementations will beapparent to those skilled in the art. The various aspects andimplementations disclosed herein are for purposes of illustration andare not intended to be limiting, with the true scope and spirit beingindicated by the following claims. Other implementations may beutilized, and other changes may be made, without departing from thespirit or scope of the subject matter presented herein. It will bereadily understood that the aspects of the present disclosure, asgenerally described herein, and illustrated in the figures, can bearranged, substituted, combined, separated, and designed in a widevariety of different configurations.

What is claimed is:
 1. A method, comprising: generating, using a lightdetection and ranging (LIDAR) sensor coupled to a vehicle, a point cloudcomprising a three-dimensional map of points indicative of locations ofreflective features in an environment; capturing, using an image sensorcoupled to the vehicle, an image of a portion of the environment,wherein the image comprises a plurality of composite pixels, eachcomposite pixel being associated with: one or more intensity valuesobtained by a first element of the image sensor, and one or moreintensity values obtained by at least a second element of the imagesensor, wherein a distance from the second element of the image sensorto the first element of the image sensor is determined in view of ascanning frequency of the image sensor; mapping one or more compositepixels in the image to one or more points in the point cloud to providecombined sensor data; and using the combined sensor data to determine adriving path for the vehicle.
 2. The method of claim 1, wherein each ofthe one or more intensity values is associated with a respectivespectral portion of light captured by the image sensor.
 3. The method ofclaim 1, wherein the combined sensor data includes spectral informationdetermined using the image sensor and distance information determinedusing the LIDAR sensor.
 4. The method of claim 1, further comprising:determining a path of motion of an object associated with the one ormore pixels based on a plurality of images captured using the imagesensor.
 5. The method of claim 4, further comprising: determining adistance to the object based on the one or more points in the pointcloud, wherein the path of motion of the object is further based on thedetermined distance to the object.
 6. The method of claim 1, furthercomprising: simultaneously rotating the LIDAR sensor and the imagesensor about an axis.
 7. A system, comprising: a light detection andranging (LIDAR) sensor configured to generate a point cloud comprising athree-dimensional map of points indicative of locations of reflectivefeatures in an environment; an image sensor configured to capture animage of a portion of the environment, wherein the image comprises aplurality of composite pixels, each composite pixel being associatedwith: one or more intensity values obtained by a first element of theimage sensor, and one or more intensity values obtained by at least asecond element of the image sensor, wherein a distance from the secondelement of the image sensor to the first element of the image sensor isdetermined in view of a scanning frequency of the image sensor; and acontroller configured to map one or more composite pixels in the imageto one or more points in the point cloud to provide combined sensordata.
 8. The system of claim 7, wherein the LIDAR sensor and the imagesensor are coupled to a vehicle.
 9. The system of claim 8, furthercomprising: a navigation and pathing system configured to use thecombined sensor data to determine a driving path for the vehicle. 10.The system of claim 7, wherein the combined sensor data includesspectrum information determined using the image sensor and distanceinformation determined using the LIDAR sensor.
 11. The system of claim7, wherein the controller is further configured to determine a path ofmotion of an object associated with the one or more pixels based on aplurality of images captured using the image sensor.
 12. The system ofclaim 11, wherein the path of motion of the object is further based on adistance to the object determined based on the one or more points in thepoint cloud.
 13. The system of claim 7, further comprising: an actuatorthat simultaneously rotates the LIDAR sensor and the image sensor aboutan axis.
 14. A non-transitory computer readable medium having storedtherein instructions executable by a computing device to cause thecomputing device to perform operations, wherein the operations comprise:generating, using a light detection and ranging (LIDAR) sensor coupledto a vehicle, a point cloud comprising a three-dimensional map of pointsindicative of locations of reflective features in an environment;capturing, using an image sensor coupled to the vehicle, an image of aportion of the environment, wherein the image comprises a plurality ofcomposite pixels, each composite pixel being associated with: one ormore intensity values obtained by a first element of the image sensor,and one or more intensity values obtained by at least a second elementof the image sensor, wherein a distance from the second element of theimage sensor to the first element of the image sensor is determined inview of a scanning frequency of the image sensor; mapping one or morecomposite pixels in the image to one or more points in the point cloudto provide combined sensor data; and using the combined sensor data todetermine a driving path for the vehicle.
 15. The non-transitorycomputer readable medium of claim 14, wherein each of the one or moreintensity values is associated with a respective spectral portion oflight captured by the image sensor.
 16. The non-transitory computerreadable medium of claim 14, wherein the combined sensor data includesspectral information determined using the image sensor and distanceinformation determined using the LIDAR sensor.
 17. The non-transitorycomputer readable medium of claim 14, wherein the operations furthercomprise: determining a path of motion of an object associated with theone or more pixels based on a plurality of images captured using theimage sensor.
 18. The non-transitory computer readable medium of claim17, wherein the operations further comprise: determining a distance tothe object based on the one or more points in the point cloud, whereinthe path of motion of the object is further based on the determineddistance to the object.